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ABSTRACT 


The  Recruit  Assessment  Program  (RAP)  study  was  begun  in  June  2001  to  assess  the 
feasibility  of  collecting  computerized  comprehensive  baseline  health  information  from  new 
recruits,  including  data  on  demographics,  medical  history,  psychosocial  history,  substance  abuse, 
and  family  history.  Past  evaluations  of  from  the  Adverse  Childhood  Experiences  (ACE)  study 
have  found  a  relationship  between  such  experiences  and  adverse  adult  health  outcomes, 
including  substance  abuse,  attempted  suicide,  and  depression.  Thus,  the  ACE  questions  were 
added  to  the  RAP  questionnaire  in  June  2002.  Focus  group  experiences  with  ACE  questions  on 
the  RAP  instrument,  completed  by  Marine  Corps  recruits  in  San  Diego  and  Army  recruits  in  Fort 
Jackson,  revealed  no  specific  concerns.  In  extensive  post- focus  group  implementation  of  RAP 
(>100,000  questionnaires  completed  overall),  ACE  questions  also  appeared  well  accepted  and 
completed  without  problems.  Reliability  metrics,  overall,  indicate  that  ACE  questions  were 
likely  assessing  what  they  were  intended  to  assess  among  RAP  respondents.  Reasonably  strong 
concordance  was  noted  for  internal  reliability  within  the  instrument.  Test-retest  results  were 
robust,  especially  when  the  instrument  was  completed  with  only  a  short  time  interval  between 
testing.  Results  from  these,  and  other  analyses,  will  be  helpful  to  DoD  policymakers  considering 
the  value  of  ACE  questions  in  future  military  questionnaires. 
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INTRODUCTION 


As  highlighted  by  the  Persian  Gulf  War  of  1990-1991,  there  is  a  need  to  obtain  accurate 
baseline  health  data  from  all  military  service  members.  The  Recruit  Assessment  Program  (RAP) 
study  was  begun  in  June  2001  to  assess  the  feasibility  of  collecting  computerized  comprehensive 
baseline  health  information  from  new  recruits,  including  data  on  demographics,  medical  history, 
psychosocial  history,  substance  abuse,  and  family  history.1,2 

The  RAP  questionnaire  was  developed  by  public  health  officials,  clinicians,  and 
researchers  from  the  Department  of  Defense  (DoD),  Veterans  Health  Administration,  and 
Department  of  Health  and  Human  Services.  Questions  were  derived  from  standardized  survey 
instruments,  including  the  Alcohol  Use  Disorders  Identification  Test,3,4  the  National 
Comorbidity  Study,5,6  the  Adverse  Childhood  Experiences  (ACE)  Study,7'11  the  Childhood 
Trauma  Questionnaire,12,13  and  the  Conflict  Tactics  Scales.14  Past  evaluations  of  ACE  have 
found  a  relationship  between  such  experiences  and  adverse  adult  health  outcomes,  including 
substance  abuse,  attempted  suicide,  and  depression.  '  Thus,  the  ACE  questions  were  added  to 
the  RAP  questionnaire  in  June  2002,  when  the  second  version  of  the  RAP  questionnaire  was 
introduced.  A  third  version  of  the  questionnaire  was  introduced  in  October  2002,  which  was 
different  from  prior  versions  primarily  in  the  language  of  the  informed  consent  and  notice  of 
privacy  practices.  A  fourth  version  of  the  RAP  questionnaire  was  implemented  in  March  2003, 
and  is  currently  in  use  among  Marine  recruits  in  San  Diego.  It  differs  from  previous  versions  in 
that  it  asks  additional  education  questions,  such  as  history  of  homeschooling,  school  suspension, 
learning  disabilities,  and  family  socioeconomic  status  (see  Appendix  A). 


FOCUS  GROUP  TESTING 

Focus  group  tests  were  conducted  between  2000  and  2002  among  Marine  Corps  recruits 
at  Marine  Corps  Recruit  Depot,  San  Diego,  CA,  and  between  2002  and  2004  among  Army 
recruits  at  U.S.  Army  Training  Center,  Fort  Jackson,  SC.  The  initial  RAP  survey  was  tested  by 
198  male  recruits  in  small  focus  groups  of  10  to  20  recruits.  Recruits  were  given  the  opportunity 
to  comment  on  difficulties  encountered  answering  individual  questions,  as  well  as  to  discuss  the 
length  and  acceptability  of  the  overall  questionnaire.  The  original  questionnaire  required  60 
minutes  to  complete  and  was  1 7  pages  in  length.  Recruit  feedback  from  the  focus  groups  was 
used  to  shorten  the  length  of  the  questionnaire,  so  that  the  current  version  takes  about  30  minutes 
to  complete  and  is  12  pages  in  length.  Questions  were  simplified,  and  revised  questions 
improved  readability  and  comprehension,  especially  on  family  history  issues.  Since  a  large 
number  of  recruits  reported  not  coming  from  two-parent  homes,  questions  were  clarified  to  refer 
to  either  “biological  father”  or  “the  father  who  raised  you”  and  “biological  mother”  or  “the 
mother  who  raised  you.” 

In  all  focus  group  testing,  no  specific  concerns  were  voiced  regarding  the  ACE  questions. 
It  should  be  noted  that  Marine  Corps  training  in  San  Diego  includes  only  male  recruits,  while 
Army  training  in  Fort  Jackson  includes  both  men  and  women.  It  is  not  clear  whether  opinions  of 
focus  groups  from  these  two  populations  can  be  generalized  to  all  military  recruits.  There  may 
also  be  concern  that,  although  recruits  were  forthcoming  about  many  questionnaire  concerns, 
they  may  not  have  been  candid  about  ACE  concerns  in  focus  group  settings. 
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POST-FOCUS  GROUP  EXPERIENCES 


Nearly  80,000  RAP  surveys  were  collected  from  Marines  in  San  Diego  between  June 
2001  and  January  2006,  representing  99%  of  recruits  entering  training  at  this  site.  Again, 
throughout  this  period,  no  specific  concerns  have  been  voiced  regarding  the  ACE  questions. 

Question-by-question  completion  rates  have  been  monitored  for  the  RAP  questionnaire 
(Figure  1),  allowing  examination  of  questions  that  are  commonly  skipped  or  missed.  In  San 
Diego,  the  10  most  frequently  skipped/missed  questions  on  the  RAP  instrument  range  from  6.6% 
to  8.5%  missing  (Table  1),  with  the  most  commonly  skipped  question  being  at  the  end  of  the 
questionnaire:  “During  the  year  prior  to  entering  the  military,  [were  you]  limited  in  any  kind  of 
work  or  other  daily  activities  as  a  result  of  your  physical  health?”  This  question  originates  from  a 
standard  instrument,  the  SF-36  health  survey15  and  presumably  is  slightly  more  difficult  to 
interpret  for  some  recruits.  Note  the  slight  drop  in  completion  rates  near  the  end  of  the 
questionnaire,  as  seen  in  Figure  1,  likely  representing  a  fatigue  factor  or  lack  of  time  to  complete. 

The  ACE  questions  appear  on  pages  10  and  1 1  of  the  current  12-page  instrument,  and 
they  are  not  among  the  top  10  frequently  skipped/missed  questions.  Completion  rates  in  San 
Diego  range  from  96.1%  for  the  question,  “How  often  did  a  parent  or  other  adult  living  in  your 
home  push,  grab,  shove,  slap,  or  throw  something  at  you?,”  to  95.3%  for  the  question,  “How 
often  did  a  parent  or  adult  living  in  your  home  swear  at  you,  insult  you,  or  put  you  down?”  (see 
Table  2).  The  completion  rate  figure  (Figure  1)  also  illustrates  that  the  questions  preceding  the 
ACE  questions — which  ask  personal  history  about  traffic  tickets,  seat  belt  use,  age  of  first 
intercourse,  condom  use,  and  sexually  transmitted  disease  history — all  have  lower  completion 
rates  than  the  ACE  questions.  This  would  imply  that  the  ACE  questions  were  not  as  difficult  to 
complete  for  most  recruits  as  the  preceding  questions. 

At  Fort  Jackson,  35,492  Army  trainees  completed  the  RAP  questionnaire  between 
October  2002  and  May  2004.  This  represents  74%  of  the  trainee  population  over  that  time 
period.  Approximately  40%  of  Fort  Jackson  RAP  respondents  were  women.  Question-by 
question  completion  rates  from  Fort  Jackson  (October  2002  through  September  2003)  show  a 
trend  for  increasing  nonresponse  rates  (men  and  women)  throughout  the  final  section  of  the 
survey,  suggesting  that  recruits  had  insufficient  time  to  complete  the  survey  or  were  fatigued 
(Figure  2). 16  The  ACE  questions  are  in  this  final  section,  but  did  not  have  the  highest 
nonresponse  rates  in  the  section.  In  the  first  sections  of  the  survey,  the  Fort  Jackson  findings 
show  that  questions  regarding  occupational  exposures,  number  of  alcohol  drinks  consumed  in  a 
typical  day  of  drinking,  and  whether  the  recruit  was  right-  or  left-handed  had  higher  rates  of 
nonresponse  than  other  questions  on  the  questionnaire,  similar  to  findings  among  Marines  in  San 
Diego.  The  most  commonly  skipped  questions  at  Fort  Jackson,  with  13.3%  missing,  were  again 
located  at  the  end  of  the  questionnaire:  “During  the  year  prior  to  entering  the  military,  [were  you] 
limited  in  any  kind  of  work  or  other  daily  activities  as  a  result  of  your  physical  health?”  and 
“During  the  year  prior  to  entering  the  military  [did  you  not]  do  work  or  other  activities  as 
carefully  as  usual  as  a  result  of  your  emotional  health?” 

In  addition  to  completion  rate  analysis,  the  RAP  team  at  Fort  Jackson  provided  unique 
insight  into  the  questionnaire’s  acceptability  through  a  separate  mechanism.  The  team  distributed 
an  anonymous  survey  (Appendix  B)  to  collect  trainees’  comments  on  the  RAP  questionnaire,  and 
specifically  which  questions  were  “skipped”  and  which  questions  were  “difficult  to  answer.” 

Of  the  5033  Army  trainees  invited  to  complete  the  RAP  questionnaire  in  October- 
November  2002,  3189  (63.4%)  completed  RAP.  Of  these  participants,  1779  (55.8%)  completed 
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the  subsequent  anonymous  survey.  Comments  from  participants  included  that  they  found  RAP 
interesting  (21.9%),  had  problems  with  certain  questions  (13.5%),  found  some  questions  too 
personal  (12.3%),  and  participated  to  help  other  soldiers  (9.0%).  Only  412  (22.3%)  of  the  1844 
who  did  not  complete  the  RAP  questionnaire  completed  an  anonymous  survey.  These  trainees 
revealed  that  they  elected  not  to  participate  because  RAP  was  too  personal  (21.7%),  they  were 
concerned  about  confidentiality  (20.4%),  they  were  not  interested  (1 1.3%),  or  they  did  not  fully 
understand  the  purpose  (10.2%).  When  all  1779  survey  respondents  were  asked  which  questions 
were  “difficult  to  answer,”  the  ACE  questions  were  not  cited  in  the  top  10  (Table  3).  When  asked 
which  questions  were  “skipped,”  1 1  trainees  cited  the  ACE  question  on  sexual  abuse,  making  it 
the  9th  most  cited  in  this  anonymous  survey  sample  (Table  4).  Overall,  the  Army  trainees  from 
this  survey  found  RAP  questions  on  alcohol,  smoking,  and  sex  more  “difficult”  or  [likely  to  be] 
“skipped”  than  questions  on  ACE. 


RELIABILITY  ANALYSES 

Concordance  coefficients  (Kappa  statistics)  are  estimates  of  reliability;  they  quantify  the 
reproducibility  of  the  same  variable  measured  more  than  once.17,18  Kappa  values  of  <0.2  are 
usually  interpreted  as  showing  poor  agreement;  values  between  0.2  and  0.4  indicate  fair 
agreement;  values  between  0.4  and  0.6  indicate  moderate  agreement;  values  between  0.6  to  0.8 
indicate  good  or  substantial  agreement;  and  values  and  between  0.8  and  1.0  indicate  excellent 
agreement.  Two  kinds  of  reliability  analyses  have  been  performed  using  RAP  questionnaires 
from  Marine  recruits  in  San  Diego. 

Table  5  illustrates  reliability  of  a  questionnaire  concept,  asked  two  different  ways  on  the 
same  instrument  completed  by  the  same  respondent.  This  is  also  referred  to  as  “internal 
reliability.”  Note  that  there  are  two  questions  on  sexual  abuse  that  may  be  compared  in  RAP: 
“How  often  did  an  adult  ever  touch  you  sexually  or  try  to  make  you  touch  them  sexually?”  and 
“Did  any  of  the  following  events  EVER  happen  to  you  in  your  entire  life:  You  were  raped 
(someone  forced  you  to  have  sex  against  your  will).”  These  questions  are  about  different,  but 
clearly  related,  experiences.  The  Kappa  statistic  indicates  poor  internal  reliability,  but  the  overall 
concordance  is  high.  These  results  are  not  unexpected,  given  the  difference  between  the 
questions  and  low  endorsement  rates.  In  contrast,  Kappa  statistics  show  increasing  internal 
reliability  for  questions  about  family  history  of  mental  illness,  family  history  of  alcohol  abuse, 
and  personal  history  of  sexual  intercourse  (Table  5).  These  results  reflect  a  closer  relationship 
between  the  content  of  the  two  questions  compared,  as  well  has  higher  endorsement  rates  of  the 
questions. 

Table  6  illustrates  the  more  traditional  use  of  Kappa  analyses,  comparing  reproducibility 
of  the  same  questions  from  the  same  respondents  on  two  different  occasions.  These  are  also 
called  “test-retesf  ’  results.  Figure  2  demonstrates  that  overall  Kappa  statistics  were  strong  for 
test-retesting  of  RAP  at  both  1  day  and  10  weeks  apart.  Table  6  shows  the  test-retest  results  for 
specific  ACE  questions  among  252  Marine  recruits,  10  weeks  apart.  Kappa  values  ranged  from 
fair  (0.29-0.30)  for  physical  neglect  and  emotional  neglect,  to  moderately  good  (0.40-0.52)  for 
sexual  abuse,  family  history  of  mental  illness,  physical  abuse,  emotional  abuse,  domestic 
violence,  and  family  history  of  alcohol  problems.  Overall  test-retesting  showed  shifting  of 
responses  in  both  directions.  There  was  a  small  tendency  for  responses  later  in  recruit  training  to 
reflect  less  history  of  ACE,  but  this  was  inconsistent  across  all  ACE  categories.  When  the  time 
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interval  of  test-retest  of  questionnaires  was  shorter,  Kappa  values  for  all  RAP  questions  were 
much  higher.  Kappa  coefficients  tend  to  be  low  for  questions  in  which  rarer  events  are  assessed. 
For  example,  the  number  of  male  Marine  Corps  recruits  reporting  history  of  sexual  abuse  on  the 
RAP  questionnaire  tends  to  be  low,  with  a  frequency  of  1 .4% 19  since  RAP  was  implemented  in 
San  Diego  in  2001. 


POTENTIAL  FUTURE  ANALYSES 

RAP  data  from  Army  recruits  in  Fort  Jackson,  SC,  remain  to  be  analyzed  for  internal 
reliability.  In  addition,  data  from  a  larger  sample  of  post-RAP  anonymous  surveys  may  be 
available  to  add  to  previous  analyses. 

A  new  study  has  been  designed  to  assess  test-retest  reliability  of  RAP  questionnaire  data 
when  completed  at  Military  Entrance  Processing  Stations  (MEPS)  compared  with  initial  recruit 
training.  This  study  may  be  important  for  understanding  the  general  reliability  of  ACE  questions 
and  the  special  circumstances  that  may  affect  responses  at  MEPS  and  recruit  training.  The  study 
will  be  vital  if  a  RAP-like  instrument  is  ever  implemented  DoD-wide  at  MEPS. 

Finally,  and  perhaps  most  importantly,  studies  are  under  way  to  determine  the 
relationship  between  ACE  histories  among  recruits  who  completed  RAP  and  subsequent  health 
outcomes  of  military  importance.  Of  high  interest  is  a  complex  study  under  way  among  Marines 
to  assess  whether  ACE  is  a  risk  factor,  or  resiliency  marker,  for  development  of  post-traumatic 
stress  disorder  after  combat  deployment.  Full  discussion  of  such  ACE  analyses  is  outside  the 
scope  of  this  paper. 


CONCLUSIONS 

Focus  group  experiences  with  ACE  questions  on  the  RAP  instrument,  completed  by 
Marine  Corps  recruits  in  San  Diego  and  Army  recruits  in  Fort  Jackson,  revealed  no  specific 
concerns.  In  extensive  post- focus  group  implementation  of  RAP  (>100,000  questionnaires 
completed  overall),  ACE  questions  also  appeared  well  accepted  and  completed  without 
problems. 

The  efforts  of  the  RAP  team  in  Fort  Jackson  to  assess  specific  RAP  concerns  in  an 
anonymous  survey  are  noteworthy.  ACE  questions  were  not  singled  out  for  this  evaluation,  and 
were  not  cited  as  particularly  difficult  among  the  1779  respondents.  Although  1 1  recruits  did 
report  a  desire  to  skip  the  ACE  question  on  sexual  abuse,  this  number  was  lower  than  the  number 
who  expressed  concerns  about  other  questions,  and  much  lower  than  the  total  number  of 
respondents. 

Reliability  metrics,  overall,  indicate  that  ACE  questions  were  likely  assessing  what  they 
were  intended  to  assess  among  RAP  respondents.  Reasonably  strong  concordance  was  noted  for 
internal  reliability  within  the  instrument.  Test-retest  results  were  robust,  especially  when  the 
instrument  was  completed  with  only  a  short  time  interval  between  testing. 

It  is  hoped  that  results  from  these,  and  other  analyses,  will  be  helpful  to  DoD 
policymakers  considering  the  value  of  ACE  questions  in  future  military  questionnaires. 
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Figure  1.  Question-specific  completion  rates  of  the  RAP  questionnaire,  June  2002-November  2005,  Marine  Corps  Recruit 
Depot,  San  Diego  (N=60,008). 
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Table  1.  Percentage  Missing  for  the  Top  10  Questions  Missed  by  Respondents  (N=60,008)  of  the  RAP  Questionnaire  at  Marine 
Corps  Recruit  Depot,  San  Diego,  June  2002-November  2005 


No. 

Question 

%  Missing 

1 

During  the  year  prior  to  entering  the  military  did  you  have  any  of  the  following  problems  as  a  result  of  your  PHYSICAL  health? 
Were  limited  in  any  kind  of  work  or  other  daily  activities 

8.5 

2 

During  the  year  prior  to  entering  the  military  did  you  have  any  of  the  following  problems  as  a  result  of  any  EMOTIONAL 
problems  (such  as  feeling  depressed  or  anxious)?  Didn’t  do  work  or  other  activities  as  carefully  as  usual 

8.4 

3 

Are  you  mainly  right  or  left  handed? 

7.8 

4 

If  you  have  not  had  a  medical  checkup  in  the  past  5  years,  other  than  exams  done  for  military  entrance,  why  not? 

7.8 

5 

How  much  time  did  you  feel  tired  or  worn  out?  (during  the  past  year) 

7.3 

6 

If  you  have  not  had  a  dental  checkup  in  the  past  5  years,  other  than  exams  done  for  military  entrance,  why  not? 

7.0 

7 

During  the  year  prior  to  entering  the  military  did  you  have  any  of  the  following  problems  as  a  result  of  any  EMOTIONAL 
problems  (such  as  feeling  depressed  or  anxious)?  Accomplished  less  than  you  would  like 

6.8 

8 

Did  you  have  difficulty  doing  activities  involving  concentration  and  thinking?  (during  the  past  year) 

6.7 

9 

During  the  year  prior  to  entering  the  military  did  you  have  any  of  the  following  problems  as  a  result  of  your  PHYSICAL  health? 
Accomplished  less  than  you  would  like 

6.6 

10 

How  much  time  did  you  feel  downhearted  and  blue?  (during  the  past  year) 

6.6 
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Table  2.  Percentage  Missing  for  ACE  Questions  by  Respondents  (N=60,008)  of  the  RAP  Questionnaire  at 
Marine  Corps  Recruit  Depot,  San  Diego,  June  2002-November  2005 


ACE  Category 

Question 

%  Missing 

Physical  neglect 

There  was  someone  to  take  care  of  you  and  protect  you. 

4.4 

Emotional  neglect 

You  felt  loved 

4.6 

Emotional  abuse 

How  often  did  a  parent  or  adult  living  in  your  home  swear  at  you, 
insult  you,  or  put  you  down? 

4.7 

Physical  abuse 

How  often  did  a  parent  or  other  adult  living  in  your  home  push, 
grab,  shove,  slap,  or  throw  something  at  you? 

3.9 

Domestic  violence 

How  often  did  a  parent  or  other  adult  living  in  your  home  push, 
grab,  shove,  slap,  or  throw  something  at  each  other? 

4.1 

Sexual  abuse 

How  often  did  an  adult  touch  you  sexually  or  try  to  make  you  touch 
them  sexually? 

4.5 

Family  history  of 
mental  illness 

Did  you  live  with  someone  who  was  depressed  or  mentally  ill? 

4.3 

Family  history  of 
alcohol  problems 

Did  you  live  with  someone  who  was  a  problem  drinker  or  alcoholic? 

4.4 
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Figure  2.  Question-specific  completion  rates  of  the 
(N=l 7,959). 
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Table  3.  Top  10  RAP  Questions  Fort  Jackson,  SC,  Participants  Found  “Difficult  to 
Answer”  and  Reasons  for  Difficulty  for  October-November  2002  (N=1779) 


No. 

Question 

Number  of 
recruits  who 
reported 
question  was 
difficult  to 

answer 

Examples  of  reasons  why  found 
question  difficult 

1 

How  many  years  have  you  been 
drinking  alcoholic  beverages  on  a 
regular  basis? 

23 

I  didn’t  know  if  I’d  be  in  trouble  or 

not. 

What  does  “regular  basis”  mean? 

My  answer  was  not  one  of  the  choices. 

2 

During  the  past  year,  how  many  drinks 
containing  alcohol  did  you  have  on  a 
typical  day  of  drinking? 

14 

What  is  “a  typical  day  of  drinking”? 

It  makes  it  seem  like  I  drink  everyday 
&  I  don’t. 

3 

Did  you  or  your  partner  use  a  condom 
the  last  time  you  had  sex? 

14 

Question  is  way  too  personal. 

Options  were  not  good  choices. 

Felt  uncomfortable  answering. 

4 

How  old  were  you  when  you  had 
sexual  intercourse  for  the  first  time? 

13 

Question  is  way  too  personal. 

Felt  uncomfortable  answering. 

5 

Have  you  ever  tried  to  quit  smoking 
cigarettes? 

13 

“I  have  never  smoked”  is  not  one  of 
the  choices. 

6 

At  what  age  did  you  first  start  smoking 
regularly  (meaning,  you  smoked  on 
most  days)? 

13 

“Never  smoked”  should  be  an  option. 

7 

How  many  years  did  you  smoke  more 
than  3  cigarettes  on  most  days? 

12 

“Never  smoked”  should  be  an  option. 

8 

Have  you  ever  been  told  by  a  doctor  or 
nurse  that  you  had  a  sexually 
transmitted  disease  or  STD  (like 
chlamydia,  gonorrhea,  genital  herpes, 
or  syphilis?) 

12 

Very  personal  question. 

Concerned  about  confidentiality. 

9 

Is  the  father  who  raised  you  alive? 

12 

Typographical  error  in  answer  choices 
(said  “she”  instead  of  “he”).  Note: 

This  was  corrected. 

10 

During  the  year  prior  to  entering  the 
military,  did  you  have  any  of  the 
following  problems  as  a  result  of  any 
emotional  problems  (such  as  feeling 
depressed  or  anxious)? 

a.  accomplished  less  than  you  would 
like 

b.  didn’t  do  work  or  other  activities  as 
carefully  as  usual 

12 

Wording  is  confusing. 

I  did  not  understand  the  choices  for  the 
question. 

Source:  Chervak  MC,  Hauret  K,  Koth  K,  Darakjy  S,  Kenyon  M,  Canada  S,  Jones  B.  The  value  of  understanding  your 
survey  population:  lessons  from  the  Recruit  Assessment  Program  at  Ft  Jackson.  Poster  presented  at  6th  Annual  Army 
Force  Health  Protection  Conference,  August  1 1-17,  2003,  Albuquerque,  New  Mexico. 
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Table  4.  Top  10  RAP  Questions  Fort  Jackson,  SC,  Participants  “Skipped  or  Chose 
Not  to  Answer”  and  Reasons  for  Skipping  for  October-November  2002  (N=1779) 


Question 

Number  of 
recruits  who 
skipped  or 
chose  not  to 

answer 

question 

Examples  of  reasons  why  skipped  or 
chose  not  to  answer  question 

1 

How  old  were  you  when  you  had  sexual 
intercourse  for  the  first  time? 

34 

Felt  that  it  was  an  inappropriate 
question;  uncomfortable  answering. 

2 

Did  you  or  your  partner  use  a  condom  the 
last  time  you  had  sex? 

29 

Question  was  too  personal. 

Do  not  wish  to  disclose. 

3 

Have  you  ever  been  told  by  a  doctor  or 
nurse  that  you  had  a  sexually  transmitted 
disease  or  STD  (like  chlamydia, 
gonorrhea,  genital  herpes,  or  syphilis?) 

18 

Did  not  feel  comfortable  answering 
question. 

Too  personal. 

4 

During  the  past  year,  how  many  drinks 
containing  alcohol  did  you  have  on  a 
typical  day  of  drinking? 

17 

My  answer  was  not  one  of  the  choices. 

I  do  not  typically  drink  every  day. 

Do  not  wish  to  disclose. 

5 

How  many  years  have  you  been  drinking 
alcoholic  beverages  on  a  regular  basis? 

16 

My  answer  was  not  there 
(occasionally). 

What  is  considered  “regular”? 

6 

At  what  age  did  you  first  start  smoking 
regularly  (meaning,  you  smoked  on  most 
days)? 

14 

“Don’t  smoke”  is  not  an  answer 
option. 

I  did  not  feel  like  giving  that 
information. 

7 

How  many  years  did  you  smoke  more 
than  3  cigarettes  on  most  days? 

12 

“Don’t  smoke”  or  “not  applicable”  is 
not  an  answer  option. 

8 

When  you  were  smoking  regularly,  how 
many  packs  did  you  smoke  each  day? 

11 

“Don’t  smoke”  or  “not  applicable”  is 
not  an  answer  option. 

9 

How  often  did  an  adult  ever  touch  you 
sexually  or  try  to  make  you  touch  them 
sexually? 

11 

Too  personal.  None  of  your  business. 

Didn’t  feel  comfortable  answering. 

I’m  not  certain  how  this  is  relevant  to 
providing  health  care. 

10 

Have  you  ever  tried  to  quit  smoking 
cigarettes? 

10 

“Don’t  smoke”  or  “not  applicable”  is 
not  an  answer  option. 

Source:  Chervak  MC,  Hauret  K,  Koth  K,  Darakjy  S,  Kenyon  M,  Canada  S,  Jones  B.  The  value  of  understanding  your 
survey  population:  lessons  from  the  Recruit  Assessment  Program  at  Ft  Jackson.  Poster  presented  at  6th  Annual  Army 
Force  Health  Protection  Conference,  August  1 1-17,  2003,  Albuquerque,  New  Mexico. 
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Table  5.  Internal  Reliability  of  ACE  Questions  on  the  RAP  Questionnaire  from  Marine  Corps  Recruit  Depot,  San  Diego: 
Comparing  Two  Different  Questions  from  the  Same  Respondent  and  Questionnaire 


Category 

Question  1 

Question  2 

N 

Kappa 
(95%  Cl) 

% 

Concordance* 

Question 

Responses 

Question 

Responses 

Sexual 

abuse 

How  often  did  an  adult 
ever  touch  you  sexually  or 
try  to  make  you  touch 
them  sexually? 

Never, 
Once/twice, 
Sometimes, 
Often,  Very 
often* 

You  were  raped  (someone 
forced  you  to  have  sex 
against  your  will). 

Yes,  No* 

56,560 

0.12 

(0.10-0.15) 

98.1 

Family 
history  of 
mental 
illness 

Did  you  live  with  someone 
who  was  depressed  or 
mentally  ill? 

Yes,  No* 

Has  your  biological  mother 
or  father  ever  had  mental  or 
emotional  problems? 

Yes,  No,  Do 
not  know 

49,463 

0.43 

(0.42-0.45) 

94.1 

Family 
history  of 
alcohol 
problems 

Did  you  live  with  someone 
who  was  a  problem 
drinker  or  alcoholic? 

Yes,  No* 

Has  your  biological  mother 
or  father  ever  had  alcohol 
problems? 

Yes,  No,  Do 
not  know 

50,371 

0.56 

(0.55-0.58) 

91.6 

History  of 

sexual 

intercourse 

How  old  were  you  when 
you  first  had  sexual 
intercourse  for  the  first 
time? 

I  have  never 
had  sex,  I 
have  had 
sex* 

Did  you  or  your  partner  use 
a  condom  the  last  time  you 
had  sex? 

I  have  not 
had  sex,  I 
have  had 
sex* 

72,815 

0.94 

(0.94-0.94) 

98.2 

*  Responses  were  dichotomized  into  yes/no.  Those  with  missing  and  “do  not  know”  responses  were  removed  from  the  analysis. 
The  sexual  abuse  responses  were  dichotomized  into  No  (Never  raped)  and  Yes  (all  other  responses), 
f  Concordance  is  calculated  as  the  number  of  responses  that  are  identical,  divided  by  the  total  number  of  respondents. 

*  [All  other  responses] 

Cl  =-  confidence  interval. 


Kappa  Interpretation 

0.0-0. 2  poor  agreement 

0.2-0.4  fair  agreement 

0.4-0. 6  moderate  agreement 

0.6-0. 8  good/substantial  agreement 

0. 8-1.0  excellent/greater  than  substantial  agreement 
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Table  6.  Reliability  of  ACE  Questions  on  the  RAP  Questionnaire  from  Marine  Corps  Recruit  Depot,  San  Diego:  Test-Retest 
Results  Comparing  the  Same  Question  from  the  Same  Respondent  from  2  Separate  Questionnaires,  Taken  10  Weeks  Apart 


ACE  Category 

Question  from  RAP  questionnaire 

N 

Kappa 
(95%  Cl) 

%  Concordance 

Physical  neglect* 

There  was  someone  to  take  care  of  you  and 
protect  you. 

248 

0.29 

(0.19-0.39) 

62.1 

Emotional  neglect* 

You  felt  loved 

244 

0.30 

(0.20-0.40) 

66.0 

Emotional  abuse* 

How  often  did  a  parent  or  adult  living  in  your 
home  swear  at  you,  insult  you,  or  put  you 
down? 

246 

0.45 

(0.36-0.53) 

52.0 

Physical  abuse* 

How  often  did  a  parent  or  other  adult  living  in 
your  home  push,  grab,  shove,  slap,  or  throw 
something  at  you? 

247 

0.43 

(0.34-0.53) 

65.2 

Domestic  violence* 

How  often  did  a  parent  or  other  adult  living  in 
your  home  push,  grab,  shove,  slap,  or  throw 
something  at  each  other? 

249 

0.48 

(0.35-0.61) 

78.3 

Sexual  abuse 

How  often  did  an  adult  touch  you  sexually  or 
try  to  make  you  touch  them  sexually? 

248 

0.40 

(0.00-0.94) 

98.8 

Family  history  of  mental 
illness 

Did  you  live  with  someone  who  was  depressed 
or  mentally  ill? 

248 

0.41 

(0.17-0.66) 

94.0 

Family  history  of  alcohol 
problems 

Did  you  live  with  someone  who  was  a  problem 
drinker  or  alcoholic? 

247 

0.52 

(0.36-0.68) 

89.5 

These  questions  are  based  on  a  Likert  scale  from  Never  to  Very  Often  (with  5  categories)  and  therefore  weighted  Kappa  statistics  were 
calculated  instead  of  simple  Kappa  statistics. 

t  Concordance  is  calculated  by  adding  up  those  responses  that  are  the  same  and  dividing  by  the  total  number  of  respondents. 

Cl  =  confidence  interval. 

Kappa  Interpretation 

0.0-0. 2  poor  agreement 

0.2-0. 4  fair  agreement 

0.4-0. 6  moderate  agreement 

0.6-0. 8  good/substantial  agreement 

0. 8-1.0  excellent/greater  than  substantial  agreement 
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□  10  Week  Retest  (N  =  252) 

□  1  Day  Retest  (N  =  195) 

1  y  ^  ^ 

o.9  J — I - 1  T1 — n — n — ~  =,rb^- 


Figure  3.  Section-by-section  RAP  questionnaire  reproducibility  from  Marine  Corps  Recruit  Depot,  San  Diego,  when  test- 
retest  spaced  1  day  apart,  with  excellent  overall  Kappa  of  0.92,  compared  with  test-retest  spaced  10  weeks  apart,  with 
moderate  overall  Kappa  of  0.45. 
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Appendix  A.  CURRENT  RAP  QUESTIONNAIRE  USED  AT  MARINE  CORPS  RECRUIT 
DEPOT,  SAN  DIEGO 

See  attached  Document. 
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Appendix  B.  FORT  JACKSON  SURVEY  EVALUATION  FORM 

See  attached  Document. 
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